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(57) Abstract 

The present invention relates to a method and system for accessing 
time-based information (14) based on spatial coordinate information 
obtained from a user (12). Time-based information is mapped into a 
spatial representation by mapping and transforming the timing of time- 
based information segments to spatial coordinates (10). Such spatial 
coordinates could be either one, two, or three dimensional. With this 
mapping, segments of time-based information can be associated with 
different parts of a spatial object, a spatial representation, or different 
position, of a spatial movement. The segments can be accessed (14) by 
the user by physical inputs (12) in such a fashion that the kinesthetic or 
touching memory of the user can be relied upon to re-access information 
(14) and to create a sense of the whole in the information. 
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METHOD AND SYSTEM FOR SPATIAL ACCESSING OF 
TIME-BASED INFORMATION 

Technical Field 

The present invention relates to a method and 
5 system for accessing time-based information based on 
spatial coordinate information obtained from a user. 

Background Art 

Absorbing and processing time-based 
information is more difficult than absorbing spatially 

10 distributed information. This is # in part, due to the 
difficulty in creating a "sense of the whole" from time- 
based information. For instance, it is a common 
experience that visual reading of printed text is much 
easier and faster than listening to recorded speech. 

15 The focus of the eye (fovea vision) can move quickly 
from one place to another; the peripheral vision fuzzily 
views a large area around the fovea so a reader knows 
where the fovea vision is with respect to the rest of 
the printed text (e.g., the rest of page, the rest of 

20 the sentence, the rest of the word) at any given moment. 

It is the peripheral vision that enables the fovea 
vision to locate quickly the places the reader wants to 
examine. Therefore, visual reading can be thought of as 
"instant arbitrary accessing". 

25 In contrast the conventional devices which 

people use to access time-based information, for 
instance, an audio and/or video player, only display 
such information sequentially (e.g., playing back 
recorded audio or video frames) . In other words, the 

30 conventional devices present time-based information only 
along a time axis. Since time is sequential, at any 
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moment, the person who is accessing such information is 
exposed to- only a small segment of the information. 
Given that a human's temporal memory is short and has 
poor resolution of time, it is difficult for users to 
5 determine the relative timing mark of a particular 
information segment with respect to the entire piece of 
information being displayed or accessed. 

Even if the underlying devices allow users to 
quickly move from one part of the speech to another 
10 (e.g., a digital storing device that allows random 

access) , a listener normally would not do so, simply 
because he/she doesn't know where he/she is with respect 
to the entire information at any moment, thus he/she 
doesn't know where to move. Thus the difficulty of 
15 accessing time-based information stems chiefly from the 
lack of the capability to generate a sense of the whole 
due to the combination of characteristics of human 
sight, hearing, memory, and the way conventional devices 
work. This temporal limitation makes it much more 
difficult to construct a general outlook of the set of 
information . 



20 



If time -based information, such as recorded 
speech, could be translated precisely into printed text 
or some other spatial visual codes such as pictures, 
25 accessing the time-based information could be turned 
into reading. However, automatic speech recognition is 
far from perfect. Thus alternative ways to absorb and 
present recorded speech are needed. 

Many prior art systems have attempted to solve 
30 this problem. Most of the previous research has 
emphasized one of the following approaches : 
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1 . Condensing the information by extracting 
important (situation dependent) portions or by 
throwing out insignificant portions. This 
approach is not effective in all circumstances 
since it assumes that some of the time -based 
information, is in fact, insignificant and 
that an efficient method exists for 
prioritizing the information. 

2 . Utilizing the characteristics of the 
information, such as the amplitude of the 
waveform of a recorded speech, and structural 
patterns to visually annotate the information 
so that different segments can be easily 
distinguished- Thus this approach relies upon 
an effective automatic feature extraction 
system. This approach is also ' difficult to 
implement and not effective for all sources of 
time -based information. 

Other prior art systems have addressed the 
20 presentation of recorded speech based upon the prior 
knowledge of the structure of the underlying text. An 
example of one such system is the multimedia package 
"From- Alice to Ocean" marketed by Apple Computers. This 
package presents the story of a woman's journey across 
25 the Australian desert. The text of the story, as read 
by the author, has been stored in convenient segments. 
These segments of text are accessed by a user along with 
corresponding visual images based upon visual cues 
displayed on a computer screen. 

30 A second prior art system of this kind is a 

CD-ROM game/reader, "Just Grandma and Me, " marketed by 
Broderbund. This package allows a user to play back 
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prerecorded spoken words corresponding to the words of 
a story displayed on a computer screen. 

These systems and their underlying methods do 
not address the problem of accessing time-based 
5 information without the additional a priori information 
provided by the presence of the text. If the underlying 
text is known, the solution to the problem is 
trivialized since the text itself could be used to 
directly access the information, e.g. directly reading 
10 or text -to -speech translation. This is not an effective 
approach for time-based information which lacks a priori 
access to a textual interpretation or other similar 
script . 

Summary Of The Invention 

!5 It is thus an object of the present invention 

to provide a method and system for permitting a user to 
spatially access and play back time -based information. 

It is also an object of the present invention 
to create a sense of the whole for accessing time-based 
20 information which is analogous to what peripheral vision 
creates in reading printed text. 

It is thus a further object of the present 
invention to provide a method and system for accessing 
time -based information by converting it to spatially 
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distributed information so that the information can be 
easily accessed and browsed. 

It is thus an additional object of the present 
invention to provide a method and system for converting 
5 or mapping time -based information to spatially 
distributed information. 

Moreover, an object of the present invention 
is to provide a method and system for accessing the 
spatialized information in a way that is relevant to the 
10 time -based origins of the information. 

An additional object of the present invention 
is to provide a method and system which allow a user to 
utilize kinesthetic and tactile memories to memorize the 
locations of segments of the spatialized information and 
15 obtain a sense of the whole such that the user can 
achieve -an apparent parallel and arbitrary access of the 
information which resembles visual reading. 

A further object of the present invention is 
to provide a method and system of using physical inputs 
20 from a user which include any human body movement (e.g., 
fingers, tongue, head, shoulder, knees, toe, etc..) to 
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scan a spatialized representation of time -based 
information. 



A more specific object of the present 
invention is to provide a method and system for 
permitting a user to spatially access and play back 
time-based information, by physical inputs, in such a 
manner that the human spatial memory of the user can be 
relied upon to re-access information previously 
accessed. In this manner, the information can be 
skimmed by the user to determine portions of the 
information which warrant a more careful scanning. 

In carrying out the above objects, the present 
invention provides a method for permitting a user to 
spatially access and play back time-based information 
15 having a plurality of segments . The method includes the 
step of mapping each of the plurality of segments of the 
time— based information to a corresponding one of a 
plurality of mapped spatial coordinates. Next an input 
spatial coordinate is determined, corresponding to a 
20 first physical input of the user generated by a movement 
of a portion of the user's body to a first position. 
A first segment of the time-based information whose 
mapped spatial coordinate corresponds to the input 
spatial coordinate is then accessed, and the first 
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segment *is played back to the user. After the portion 
of the user's body has moved to a second position, a 
human spatial memory of the user can be used to return 
the portion of the user's body to the first position to 
5 re-access and play back the first segment of time-based 
information by repeating the steps above . 

In carrying out the above objects, the present 
invention further provides a method for permitting a 

10 user to spatially access and play back time-based 
information having a plurality of segments. The method 
includes assigning a sequence of corresponding timing 
. marks to the time-based information which denote the 
passage of time in the time-based information, forming 

15 the plurality of segments based on the sequence of 
timing marks, and mapping each of the sequence of timing 
marks into a corresponding one of a plurality of mapped 
spatial coordinates . 

An input spatial coordinate is determined 
20 which corresponds to a first physical input of the user 
generated by a movement of a portion of the user's body 
to a first position. A first segment of the time -based 
information whose mapped spatial coordinate corresponds 
to the input spatial coordinate is accessed. This 
25 accessing is performed by comparing the input spatial 
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coordinate to one or more of the plurality of mapped 
spatial coordinates to identify a mapped spatial 
coordinate which matches the input spatial coordinate. 
The first segment is identified as the segment which 
5 corresponds to the matched mapped spatial coordinate 
based on the corresponding timing mark. The first 
segment is then played back to the user. After the 
portion of the user's body has moved to a second 
position, a human spatial memory of - the user can be used 
10 to return the portion of the user's body to the first 
position to re-access and play back the first segment of 
time-based information by repeating the steps above. 

In carrying out the above objects, the present 
invention further provides a system for implementing 
15 each of the methods described. 

The objects, features and advantages of the 
present invention are readily apparent from . the 
following detailed description of the best mode for 
0 carrying out the invention when taken in connection with 
the accompanying drawings . 
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Brief Description Of The Drawing s 



Fig. 1 is a flow-chart representation of one 
embodiment of the method of the present invention. 

Fig. 2 is a block-diagram representation of 
5 one embodiment of the system of the present invention. 

Fig. 3 is a flow- chart representation of the 
method of transforming time-based information into 
spatial -based information of one embodiment of the 
present invention. 

10 Fig. 4 is a flow-chart representation of a 

method for accessing time -based information 
corresponding to an input spatial coordinate of one 
embodiment of the present invention. 

Fig. 5 is a block-diagram representation of an 
15 alternative embodiment of the system of the present 
invention. 

Fig. 6 is a pictorial representation of one 
embodiment of the system of the present invention which 
emphasizes a two-dimensional touch sensitive user. 
20 interface device. 
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Fig. 7 is a pictorial representation of an 
alternative two-dimensional user interface device for 
use with one embodiment of the system of the present 
invention . 

5 Figs. 8a-c present scanning patterns to be 

used with alternative two-dimensional user- interface 
devices for use with one embodiment of the system of the 
present invention. 

Figure 9 presents a spatial multiple page 
10 structure used with one embodiment of the present 
. invention. 

While the invention will be described in 
connection with the several embodiments, it will be 
understood that this description is not intended to 
15 limit the invention to these embodiments. On the 
contrary, the invention is intended to cover all 
alternatives, modifications and equivalents as may be 
included within the spirit and scope of the invention as 
defined by the appended claims . 
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Best Mode For Carrying Out The Invention 

The present invention solves the problems 
presented by the prior art in accessing time-based 
information by mapping and transforming the timing of 
5 time-based information segments to spatial coordinates. 
Such spatial coordinates could be any Cartesian or non- 
Cartesian coordinates in one, two, or three dimensional 
Euclidean or non-Euclidean space. With this mapping, 
segments of time-based information can be associated 
10 with different parts of a spatial object, a spatial 
representation, or a different position of a spatial 
movement . 

It should be understood that the term, "time- 
based information, " as used herein should be interpreted 

15 broadly to include, but not be limited to, any general 
signal or collection of data which can be expressed in 
temporal form. Examples include audio signals, whether 
analog or digital, such as speech, music, and MIDI 
signals; visual information, such as video, animation 

20 and other graphical information; and other temporal data 
such as electromagnetic signals, stock closing 
information, daily temperature data, etc. 
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Turning now to Fig. 1, a flow-chart 
representation of one embodiment of the present 
invention is presented. The timing of time-based 
information, such as audio, video, or any other time- 
5 based signal, is mapped to a set of spatial coordinates 
as shown in step 10. Various methods are possible for 
mapping the timing of time -based information to spatial 
coordinates. These include, but are not limited to: 
mapping continuous timing marks of time -based 
10 information to spatial coordinates, mapping sequential 
numbers of words (in a piece of speech) to spatial 
coordinates, and mapping other structural formations 
found in time-based information to spatial 
representations . 

15 These spatial coordinates may have any number 

of degrees of freedom. Thus, the transformation of the 
present invention may map the time-based information 
into three coordinates in a three-dimensional space, map 
the time-based information into three coordinates of 

20 position and three coordinates of orientation of a 
geometric solid in three-dimensional space, or map the 
time-based information into a set of positions and 
orientations for a series of connected joints in a 
three-dimensional space. 
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An input spatial coordinate is then determined 
based upon a physical input from the user generated by 
the movement of the user's body as shown in step 12. 
This step could be performed by sensing the position, 
5 velocity, acceleration and/or orientation of one or more 
portions or joints of the user's body. Examples include 
but are not limited to utilizing multiple finger 
positions on a position-digitizer pad that can convert 
fingers' position on the pad to a set of spatial 
10 coordinates, positions generated by a hand controller or 
tongue -controlled positioning device, or positions 
generated by an interface device which responds to head 
movement. Interface devices of this sort are commonly 
used in virtual reality applications. 

15 The determined input spatial coordinate must 

be in the same coordinate space as the mapped spatial 
coordinates of the time-based information. Thus, each 
segment of time-based information would correspond to a 
possible input spatial coordinate. 

2 0 A segment of the time -based information whose 

mapped spatial coordinate corresponds to the input 
spatial coordinate is then accessed as shown in step 14, 
and the accessed first segment of time-based information 
is then played back to the user as shown in step 16. At 
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this point the method may end, or optionally, may 
continue by determining a new input spatial coordinate 
accessing further information corresponding to the new 
input spatial coordinate and playing back this new 
5 accessed information as shown by path 18 and steps 12, 
14, and 16. By this means, after the portion of the 
user's body has been moved to a subsequent position, 
human spatial memory can be used to return the user's 
body to the first position to re-access and play back 
10 the segment of time-based information. This human 
spatial memory could be kinesthetic memory, the memory 
that memorizes spatial positions of human physical 
movement through sensing muscle tension. The term 
"Human Spatial Memory" should be broadly constituted to 
15 also cover other spatial memories of the user such as 
touching memory of the user, the memory that memorizes 
spatial locations on a portion of a human body (e.g., a 
palm) where skin is stimulated by touching. 

In this fashion, the sense of the whole 
20 capability of peripheral vision provided in visual 
reading by human memories can be emulated by spatial 
movement and positioning of the user's body. This 
emulation may occur by utilizing a finger's kinesthetic 
memory, a palm's tactile and touching memory, a tongue's 
25 kinesthetic memory, or other types of human spatial 
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memory based upon the choice of interface device and 
based upon the choice of possible input spatial 
coordinates . 

Fig* 2 presents a block-diagram representation 
5 of one embodiment of the system of the present 
invention. The timing of time-based information 20, 
having a plurality of segments, is mapped to a set of 
spatial coordinates by timing-to-spatial converter 22. 
This converter could be implemented by a programmable 
10 logic array or discrete logic device, or . by an analog 
computational device. However, in the preferred 

embodiment, this timing-to-spatial conversion is 
performed by software or firmware running on a central 
processing unit of a computer, such as a microprocessor. 

User interface device 26 determines an input 
spatial coordinate corresponding to a physical input 
generated by the user 30. Arrow 34 represents the 
physical input from user 30 to user interface device 26. 
User interface device 26 is also capable of providing a 
sensory output or feedback 42 to user 30. This sensory 
output may take the form of some vibration, touch, 
temperature, or other sensory information which 
corresponds to the accessed time-based information and 
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which aids the user in obtaining a sense of the whole of 
the time-based information in input coordinate space. 

User interface device 2 6 in turn provides the 
determined input spatial coordinate 36 to spatial access 
5 device 24 ; This spatial access device accesses a 
segment of time -based information whose mapped spatial 
coordinate corresponds to the input spatial coordinate. 
This accessed information 3 8 is transferred from 
spatial -access device 24 to playback device 28 in order 

10 to provide a sensory output 32 to user 30. Sensory 
output 32 of playback device 28 is provided in a manner 
which emulates the form of the original form of the 
time-based information. Thus, if the original 

information is audio, the playback device plays back 

15 this audio information to the user as sound. Further, 
if the time -based information is graphical information 
such as video or animation, then the playback device 
plays back the time-based information to the user in the 
form of graphical images. These sensory outputs 32 of 

2 0 playback device 28 should be contrasted with the sensory 
output 42 of user interface device 26, whose purpose is 
to aid users in accessing the information. The sensory 
output 42 would generally be in a form which differs 
from the original form of the time-based information. 

25 Examples of such sensory output include vibration and 
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other tactile output as well as supplementary audio and 
visual cues which do not emulate the time -based 
information . 

Turning now to Fig. 3, one method is presented 
5 for performing the step of mapping the timing of the 
time-based information into a set of spatial 
coordinates, as shown in step 10 of Fig. 1. The time- 
based information is assigned a series of sequential 
timing marks as shown in step 50. These timing marks 

10 could be equally spaced in time, or unequally spaced. 

If the timing marks were unequally spaced, they could be 
. arranged based upon the structural attributes of the 
information. For example, in the case of speech, the 
timing marks could be used to set boundaries between 

15 syllables, words, sentences, paragraphs, or pages of 
spoken words thereby forming segments of time -based 
information. These timing marks are then each mapped 
into a corresponding one of a plurality of mapped 
spatial coordinates which in turn correspond to the set 

20 of possible input spatial coordinates as shown in step 
52. 

The segments of the time-based information 
which are accessed may correspond to the time -based 
information between two successive timing marks. In 
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this case, the segments are non- overlapping . This need 
not be the case for all applications. For instance, a 
segment corresponding to a given timing mark may be the 
time based information from the preceding timing mark to 
the subsequent timing mark. Further, filtering or 
windowing may be used to allow a segment to fade in at 
the beginning and/or fade out at the end. 



Turning now to Fig.- 4, a flow- chart 
representation of one embodiment of step 14, accessing 
information corresponding to an input spatial 
coordinate, as shown in Fig. l, is presented. The input 
spatial coordinate is compared to a mapped spatial 
coordinate as shown in step 54. If the input spatial 
coordinate matches the mapped spatial coordinate, then 
a segment of the time-based information is identified 
which corresponds to the matched storage spatial 
coordinate as shown in step 58. if the input spatial 
coordinate- does not match the mapped spatial coordinate, 
then a next -mapped spatial coordinate is chosen for the 
comparison as shown in step 56, and step 54 is repeated 
until a match is found. 

To summarize, the three major principles 
behind this invention are: (i) mapping the timing of 
time-based information to a set of spatial coordinates; 
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(ii) utilizing kinesthetic and touching memory to help 

construct a sense of the whole for accessing the 

information; and (iii) using various modes of 

interaction to aid users in -moving around in the 

5 information and controlling the transformation. A 
general procedure for constructing an embodiment of the 
present invention is given as follows : 



(1) Choose the form of physical movement and 
physical input to signify an input spatial 

10 coordinate. In particular, any moveable 

portion of the human body can be used to 
generate a physical input . 

(2) Decide what patterns of movement to use. 

(3) Choose a mapping from the timing marks of the 
15 time-based information to the spatial 

coordinates that the moving pattern traverses. 
At this stage, physical (e.g., electronically) 
devices are involved to sense the spatial 
movement and display the sound according to a 
20 given time -space mapping. 

Turning now to Fig. 5, a block diagram 
representation of an alternative embodiment of the 
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system of the present invention is presented. Given that 
the ear is the second best information- absorbing sensory 
organ (next to the eyes) , in this embodiment of the 
present invention, we consider using the ears to listen 
5 to sound (as opposed to converting sound into the media 
perceived by other sensory modality, e.g., tactile) . To 
move from part of spatially distributed sound to 
another, one can use any portion of the human body that 
can move spatially. One of the most convenient portions 

10 of the body is the hand, i.e., palm and fingers. 

Fingers can move quickly enough to match the speed with 
which the ears can listen. In this embodiment of the 
present invention, the kinesthetic and touching memories 
associated with one or more fingers, coupled with 

15 listening, are used to create a sense of the whole, 
similar to an "overview" capability in reading. 

This embodiment of the present invention uses 
a touch sensitive pad to provide input spatial 

20 coordinates in a two-dimensional space. The timing of 
time-based information is converted to two-dimensional 
spatialized data which is then browsed and accessed with 
fingers. Timing marks are mapped onto a page format, 
i.e., sound are distributed on the surface in rows like 

25 printed text. The fingers provide a source of 
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kinesthetic and tactile memory which allow the oiser to 
create a representation of the information. 



More particularly, the timing of audio 
information 60 is transformed into a two-dimensional 
5 spatial format by transformation device 62. This 
transformation device creates a series of timing marks 
in the audio information and maps these timing marks 
into a set of two-dimensional coordinates- A user 69 
accesses this audio information by means of finger 66. 

10 The device that senses finger position and movement is 
two-dimensional interface device 64, a touch- sensitive 
pad which generates a two-dimensional coordinate of the 
position where the hand is touching the surface. These 
coordinates are transferred from interface device 64 to 

15 transformation device 62 to determine the audio 
information which corresponds to these two-dimensional 
coordinates. This audio information is in turn 
transferred back to interface device 64, which provides 
a vibration feedback to finger 66. This vibration 

20 feedback is implemented by providing a vibratory output 
by means of an activator or speaker (not shown) attached 
to the interface device 64 . The vibration output is in 
proportion to an audio signal created from the accessed 
audio information. The accessed audio information is 

25 further provided to audio playback device 68, which 
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plays back the accessed audio information to the ears 67 
of user 69. The memory 65 of user 64 guided by touch 
and memorized position of the user's finger in "the 
whole"" allows the user to re-access a previously 
5 accessed portion of audio signal. 

Note that there are many spatial distribution 
patterns of the sound on such a surface . Examples 
include : 



(i) Playing back only the segment which 
10 is corresponding to the spatial 

location the finger is pointing to; 

(ii) Playing back a certain length of 
sound starting from a segment which 
corresponds to the spatial location 

15 of the finger ; 

(iii) Playing back a certain length of 
sound starting from the significant 
place (such as a beginning of a 
word) which is the nearest to the 

20 segment which corresponds to the 

finger location; or 

(iv) In both (ii) and (iii) , when playing 
back the sound, the sound volume 
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fades with time or is filtered in 
some form. 

Fig. 6 presents a pictorial diagram which 
represents the operation of the system of Fig. 5. Audio 
5 information 60 is stored in computer 74. Computer 74, 
in turn, assigns a sequence of timing marks which denote 
the passage of time in the audio information 60. The 
segments of audio information between each successive 
pair of timing marks are in turn mapped to a set of two- 

10 dimensional coordinates which correspond to the two- 
dimensional coordinates of touch- sensitive pads 70 of 
. two-dimensional interface device 64 . A particular two- 
dimensional coordinate is accessed by finger 66 touching 
that coordinate. This finger position, in the form of 

15 an input spatial coordinate, is fed to computer 74, 
which accesses the corresponding segment of audio 
information and plays back that information to speaker 
78, which generates- an acoustic output. Further, 
computer 74 sends the accessed segment of audio 

20 information to interface device 64, which vibrates in 
proportion to the accessed audio segment in a manner 
which is felt by finger 66. In this application, the 
finger provides a source of kinesthetic and tactile 
memory which allows the user to create a sense of the 

25 whole in the information. 
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The operation of the system of Figures 5 and 
6 is perhaps best described in terms of the following 
hypothetical application, A short presentation is given 
by a noted scholar on the subject of Dr. Martin Luther 
5 King. A tape of the lecture is obtained by a student 
and later, the student wishes to access the information 
contained in the lecture for a report which she is 
preparing . 

The taped information is sampled and stored in 
10 the computer memory. During sampling, a series of 
timing marks are stored in conjunction with the sampled 
audio information which mark the spaces between spoken 
words of the lecture. The words of the lecture are 
mapped into a two-dimensional page format based upon the 
15 timing marks . The first word is mapped into a 
coordinate corresponding to the upper right-hand corner 
of a touch sensitive pad of a user interface device. 
Subsequent words follow in order until the first row of 
the touch sensitive pad is mapped, from right to left. 
20 The next row is similarly mapped with the next set of 
words. Each of the subsequent rows are similarly mapped 
until each of the words in the presentation are mapped. 

The student can then begin to access the 
25 information by touching the touch sensitive pad. If the 
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pad is touched at a single point, the word corresponding 
to that location is played by a speaker and the pad is 
vibrated by the same audio signal in such a fashion that 
the student can feel the vibration with her finger. 

5 The entire presentation is played back 

sequentially by the student by starting her finger in 
the upper right hand corner and moving her finger along 
the first row from right to left, then the second row 
and then each subsequent row. The speed in which she 
10 moves her finger across a row governs the speed in which 
the words are played back. She browses the presentation 
at a relatively high speed to obtain an overview of the 
material . 

The student then wishes to review the portion 
15 of the presentation which dealt with Dr. Martin Luther 
King's "I have a dream" speech. She remembers the 
spatial location of this portion of the speech with her 
kinesthetic and touching memory, that is, she remembers 
this portion of the speech based upon the position that 
20 her finger was in and the feel of the vibration of the 
professor's spoken words in this area. She easily 
"homes-in" on this area and accesses this portion of the 
presentation more slowly, with a slow scanning of this 
portion of the touch sensitive pad with her finger. 
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An alternate embodiment of the two-dimensional 
interface device 64 is presented in Fig. 7. A two- 
dimensional touch sensitive pad 82 is similarly provided 
such that audio information can be accessed by the touch 
5 of a finger. However, provisions are made for visually 
highlighting and annotating portions of the information. 
Further, ways of accessing more than a single "page" of 
audio-based information is further provided. 



Very large portions of audio information 
10 provide a problem with respect to the present invention. 

A two-hour speech, for instance, might be broken up into 
several thousand segments. These segments would be 
difficult to access if all mapped onto the coordinates 
of a hand-held or even desk- top touch sensitive pad. 
15 This problem can be solved, however, by only presenting 
a subset or "page" of the audio information for 
accessing at one particular time period. The pages 
themselves could be spatially -arranged as shown in Fig. 
9. If one were currently accessing page x, the user 
20 could switch to page y by pushing the right arrow 
amongst the arrow keys 90. Similarly, if one were 
accessing page x, page z, page v, or page w, could be 
accessed by pushing the down arrow, left arrow, or up 
arrow key 90, respectively. 
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Further, segments of audio information could 
be accessed at a hierarchial * structure such that a 
level-one screen would access all of the audio 
information broken up into segments of, for instance, 
5 one minute long. Pressing the down arrow key 90 would 
change the resolution of the two-dimensional pad such 
that the one-minute segment currently being accessed 
would be in turn the entire pad broken up into a series 
of much shorter segments of audio information. 

10 Similarly, pressing the up arrow key 90 would change 
from a lower hierarchial level to a higher hierarchial 
level, such as from level-two to level-one. Any number 
of hierarchial levels could be used to access the audio 
information alternately on a microscopic or macroscopic 

15 scale. 



Referring again to Fig. 7, once a segment of 
audio information which the user deems to be important 
is accessed and played back by audio playback device 93, 
the spatial coordinates of this portion of information 

20 can be visually highlighted by the user by pressing 
highlight-on key 92. The activation of this highlight- 
on key 92 would allow the user to change the color or 
make opaque one or more segments 84 by touching those 
segments. Once the highlight-off key 92 were pressed, 

25 the touch-sensitive pad would return to accessing 
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segments of audio information based upon the touch of 
the finger, while leaving the highlighted segments 
hi ghl i gh t ed . 

In addition, important segments of text may be 
5 similarly annotated as shown in annotated block 86, 
Once the important segment of audio information were 
found, the location of this segment could be recorded by 
activating a stylus mode with stylus-on button 94. In 
the stylus mode the user could use a stylus 88 to write 
10 on the touch* sensitive screen 82 to leave a note as to 
the content of the audio information in that region . 
The user could then return to the normal audio 
information access mode by pressing the stylus-off key 
94 . 



15 In many applications the mapping of a sequence 

of timing mark into a plurality of two-dimensional 
spatial coordinates would be performed in such a maimer 
such that sequential timing marks appear in 
substantially adjacent coordinates. Turning now to Fig. 

2 0 8a, segments of audio information could be mapped in an 
order corresponding to arrow 102. The first segment 
would appear in the upper left corner, and successive 
segments would follow the zig-zagged pattern of arrow 
102 such that a finger scanned from left to right and 



WO 95/17710 PCT/US94/14622 



-29- 

then right to left one below, etc., would sequentially 
scan the segments of audio information. This way, the 
fingers don't have to "sweep" back to the left once they 
reach the end of- a line . 

Similarly, a spiral pattern such as 104 shown 
on interface device 100 of Fig. 8b could be used, or 
vertical pattern 106 shown on interface device 100 in 
Fig. 8c. One with ordinary skill in the art would 
realize that virtually any shape could be used for a 
touch- sensitive pad. Further, the shape need not be 
flat . An interface device which allows the user to 
identify a coordinate on any arbitrary surface could be 
used. For instance, the surface of a sphere could be 
used in a two-dimensional interface device- 
Furthermore, the surface of an ordinary daily-use 
object, such as a cup, chair, or a door handle, could be 
used as an interface device. 

Other applications of the present invention 
include using the spatially distributed audio track of 
2 0 a video to browse the video. Beyond this, it is 
possible to combine the accessing of spatialized audio 
with speech- to- text translation -- where it is difficult 
to translate the speech to text, use spatialized audio. 
Overall, the present invention has potential 



10 
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applications in many broad fields including multimedia 
computer software, communications, sensory access 
equipment for the handicapped, music, visual arts, 
entertainment, and education. 

While discussed in terms of a human user with 
a human memory, the present invention should be broadly 
considered to cover non-human users such as robots or 
animals capable of kinesthetic or touching memory of 
some form . 



10 While the best mode for carrying out the 

invention has been described in detail, those familiar 
with the art to which this invention relates will 
recognize the various alternative designs and 
embodiments for practicing the invention as defined by 

15 the following claims. 
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What Is Claimed Is: 

1 . A method for permitting a user to 
spatially access and play back time-based inf ormation. 
having a plurality of segments, the method comprising 
5 the steps of: 

(a) mapping each of the plurality of segments 
of the time -based information to a corresponding one of 
a plurality of mapped spatial coordinates; 

(b) determining an input spatial coordinate 
10 corresponding to a first physical input of the user 

generated by a movement of a portion of the user's body 
to a first position; 

(c) accessing a first segment of the time- 
based information which mapped spatial coordinate 

15 corresponds to the input spatial coordinate; and 

(d) playing-back the accessed first segment of 
time-based information; 

wherein f after the portion of the user's body has 
moved to a second position, a human spatial memory of 
20 the user can be used to return the portion of the body 
to the first position to re -access and play back the 
first segment of time-based information by repeating 
steps (b) , (c) , and (d) . 
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2 . The method of claim 1 wherein the step of 
mapping includes assigning a sequence of corresponding 
timing marks to the time -based information which denote 
the passage of time in the time -based information, 
5 forming the plurality of segments based on the sequence 
of timing marks, and mapping each of the sequence of 
timing marks into a corresponding one of the plurality 
of mapped spatial coordinates. 

10 3 - The method of claim 1 wherein the step of 

accessing a first segment of time-based information 
includes comparing the input spatial coordinate to one 
or more of the plurality of mapped spatial coordinates 
to identify a mapped spatial coordinate which matches 

15 the input spatial coordinate. 

4 . The method of claim 1 wherein the step of 
accessing a first segment of time-based information 
further includes identifying the f irst segment from the 
plurality of segments of time -based information by 

20 identifying the segment which corresponds to the matched 
mapped spatial coordinate based on the timing mark 
mapped to the matched mapped spatial coordinate. 

5. The method of claim 2 wherein the time- 
25 based information includes visual information. 
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6 . The method of claim 2 wherein the time- 
based information, includes audio information. 

7. The method of claim 6 wherein the audio 
information includes speech . 

5 .8, The method of claim 7 wherein the 

sequence of timing marks are determined based upon the 
structure of the speech. 

9 . The method of claim 8 wherein the 
sequence of timing marks are equally spaced in time. 

10 10. The method of claim 2 wherein the mapped 

spatial coordinates are in a Cartesian coordinate 
system. 

. u. The method of claim 2 wherein the mapped 
spatial coordinates are in a non-Cartesian coordinate 
15 system. 

12 . The method of claim 2 wherein the mapped 
spatial coordinates are in a two-dimensional space. 

13 . The method of claim 2 wherein the mapped 
spatial coordinates are in three-dimensional space. 
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14. The method of claim 12 wherein the 
mapping of the sequence of timing marks into a plurality 
of two-dimensional mapped spatial coordinates includes 
grouping a sequential subset of the sequence of timing 

5 marks into adjacent spatial coordinates. 

15. The method of claim 1 wherein the step of 
determining a spatial input coordinate includes 
determining the orientation of the portion the user's 
body. 

1° 16. The method of claim 1 wherein the step of 

determining an input spatial coordinate includes sensing 
the coordinates of a tactile pressure generated by the 
portion of the user's body on a surface. 

17. The method of claim 1 wherein the step of 
15 determining an input spatial coordinate includes sensing 

a velocity of the portion of the user's body. 

18. The method of claim 1 wherein the step of 
determining an input spatial coordinate includes sensing 
an acceleration of the portion of the user's body. 

20 19. The method of claim 1 wherein the step of 

determining an input spatial coordinate includes sensing 
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the coordinates of a temperature increase generated by 
the portion of the user's body on a surface. 

20. The method of claim 1 wherein the step of 
playing-back the accessed first segment of time-based 

5 information includes sensory output of the first segment 
in the manner which emulates the form of the original 
form of the time-based information. 

21. The method of claim 14 wherein the step 
of playing-back the accessed first segment of time -based 

10 information further includes sensory output of the first 
segment in the manner which differs in form from the 
original form of the time-based information. 

22. A system for permitting a user to 
spatially access and play back time-based information 
having a plurality of segments, the system comprising: 

a timing- to- spat i-al converter for mapping each 
of the plurality of segments of the time -based 
information to a corresponding one of a plurality of 
mapped spatial coordinates; 

a user interface device operatively associated 
with the user so as to determine a input spatial 
coordinate corresponding to a first physical input 



15 



20 
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generated by a movement of a portion of the user's body- 
to a first position; 

a spatial access device, in communication with 
the user interface device, for accessing a first segment 
5 of the time-based information whose mapped spatial 
coordinate corresponds to the input spatial coordinate; 
and 

a playback device, in communication with the 
spatial access device for playing-back the accessed 
10 first segment of time -based information; 

wherein, after the portion of the user's body 
has moved to a second position, a human spatial memory 
of the user can be used to return the portion of the 
user's body to the first position to re-access the first 
15 segment of time -based information. 

23 . The system of claim 22 wherein the 
timing- to- spatial converter includes means for assigning 
a sequence of corresponding timing marks to the time- 

20 based information which denote the passage of time in 
the time-based information, forming the plurality of 
segments based on the timing marks, and means for 
mapping each of the sequence of timing marks into a 
corresponding one of a plurality of mapped spatial 

25 coordinates. 
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24. The system of claim 22 wherein the 
spatial access device includes comparing means for 
comparing the input spatial coordinate to one or more of 
the plurality of mapped spatial coordinates to identify 

5 a mapped spatial coordinate which matches the input 
spatial coordinate . 

25. The system of claim 22 wherein the 
spatial access device further includes identifying means 
for identifying the first segment from the plurality of 

10 segments of time-based information by identifying the 
segment which corresponds to the matched mapped spatial 
coordinate based on the timing mark mapped to the 
matched mapped spatial coordinate . 

15 26. The system of claim 24 wherein the time- 

based information includes visual information. 

27. The system of claim 24 wherein the time- 
based information includes audio information. 

28. The system of claim 25 wherein the audio 
20 information includes speech. 
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29. The system of claim 28 wherein the 
sequence of timing marks are determined based upon the 
structure of the speech. 

30. The system of claim 29 wherein the 
sequence of timing marks are equally spaced in time. 

31. The system of claim 24 wherein the mapped 
spatial coordinates are in a non-Cartesian coordinate 
system. 



32 . The system of claim 24 wherein the mapped 
10 spatial coordinates are in a Cartesian coordinate 

system. 

33 . The system of claim 24 wherein the mapped 
spatial coordinates are in a two-dimensional space. 

34 . The system of claim J2A wherein the mapped 
15 spatial coordinates are in a three-dimensional space. 

35. The system of claim 33 wherein the 
mapping of the sequence of timing marks into a plurality 
of two-dimensional mapped spatial coordinates includes 
grouping a sequential subset of the sequence of timing 
20 marks into adjacent mapped spatial coordinates. 
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36. The system of claim 35 wherein the user 
interface 'device includes determination means for 
determining the orientation of the portion of the user's 
body - 

5 37. The system of claim 16 wherein the user 

interface device includes means for sensing the 
coordinates of a tactile pressure generated by the 
portion of the user's body on a surface. 

38. The system of claim 22 wherein the user 
10 interface device includes means for sensing a velocity 

of the portion of the user's body. 

39. The system of claim 22 wherein the user 
interface device includes means for sensing an 
acceleration of the portion of the user's body. 

15 40. The system of claim 22 wherein the user 

interface device includes means for sensing the 
coordinates of a temperature increase generated by the 
portion of the user's body on a surface. 

41. The system of claim 22 wherein the 
20 playback device includes means for playing-back sensory 
output of the first segment in the manner which emulates 
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the form of the original form of the time-based 
information . 

42. The system of claim 22 wherein the 
playback device further includes means for playing-back 
sensory output of the first segment in the manner which 
differs in form from the original form of the time -based 
information . 
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